Traffic replay + playbook schema - #105

Traffic replay + playbook schema

Two packages that together make a tuning claim checkable by someone who was not in
the room: gitm/traffic/ replays real production traces through a real endpoint
and ties the result back to the workload; gitm/playbook/ holds the row that says
this knob, on this model and GPU, under this traffic, moved these numbers — and
decides when that row may be applied to live traffic.

CPU-only except for firing a replay. No new dependencies. One existing extra
moves: vllm>=0.6 → vllm>=0.23.0.

PARASAIL_POC.md is the reader's entry point. This body is what changed and why.

What is measured, and what is not

Stated first because the distinction is load-bearing.

Nothing here claims a performance win. It is the machinery that would make one
checkable.

gitm/traffic/ — replay

The three traces, measured not assumed

D 6.74 vs 1.01, input p50 9,075 vs 353 — the two axes that matter separate
cleanly on real data. check_regime_axes_separate_the_traces asserts it, because
axes that cannot tell two production traces apart are decoration.

BurstGPT_3 inserts Session ID and Elapsed time at positions 1 and 2, not
appended — a positional reader takes Session ID as the model. On 5,643 real v3
rows, session id is populated on exactly the Conversation log rows and empty
on exactly the API log rows: 90% of a real v3 trace is single-shot traffic
with no conversation, so an empty one is emitted as session_id=None rather than
dropped. Dropping would have discarded 5,115 of 5,643 rows.

Real data handling, enforced rather than logged

7 named drop reasons, each firing exactly once against both a CSV and a JSONL
dirty fixture. Counts land in TraceMeta, and a trace whose counts do not
reconcile cannot be constructed — rows_read == rows_emitted + dropped in
__post_init__. A caller's own filter counts separately from bad data and still
reconciles.

The zero-token rows are real and are the majority defect: 744 of 9,382
Conversation log rows carry 0/0; all 19 API log rows are well-formed. A
property of the log, not a parse artefact.

Fidelity, against the file vLLM reads

Not against an in-memory copy — the emitter writes it, read_timed_trace
re-parses it, compare puts the two side by side:

request_count 0 · arrival_ks 0 · input_len_ks 0 · output_len_ks 0 · rate_rps 0 · burstiness 0
PASS — the pipeline preserves the trace

Every statistic exactly 0.0, on both adapters.

Fired end to end against real vLLM 0.28.0: 40/40 completed, 12.008 s against a
12.000 s trace span, confirmed by two independent clocks — vLLM's own and the
receiving server's 11.996 s across 40 POSTs.

The silent failure this exists to catch

vLLM's --timed-trace-chunk-hash-size defaults to 16; Mooncake's blocks are
512. At the default every prompt is 32× short while completed,
duration, throughput and every percentile still read perfectly. Block size
verified rather than assumed: over 1,546 Mooncake rows, max(input_length / (len(hash_ids)·512)) = 1.0 exactly — the blocks tile each prompt with no slack.

The emitter refuses to write such a file; the joiner catches one fired from a plan
built elsewhere. On the real run total_input_tokens came back 506,280 against
506,280 — at 16-token blocks it would have been ~15,821.

Two result fields are wrong, not just missing

--save-result writes 34 keys. Under --self-timed vLLM still records the CLI's
request_rate ("inf" — a string, since json.dumps cannot write bare
Infinity) and burstiness (1.0), against the trace's real 2.837 rps and D
6.74. Exactly the two axes a playbook row keys on.

Dropped with a stated reason, values kept visible, raw JSON kept whole. A new
vLLM field cannot fall out silently: unjoined_keys() fails on any key neither
kept nor deliberately dropped — it caught rtfx on its first run.

gitm/playbook/ — the row

match.py is separate from schema.py because the match semantics, not the field
list, is the part that gets argued about.

The identity split

Exact on model+revision, GPU SKU, engine+version, source_kind, concurrency
and the knob key set. Distance on the numeric regime axes, because live
traffic never lands on a measured point. Each gate is a named MatchPolicy field,
so loosening one is visible in a diff rather than an accident inside a comparison.

source_kind is asserted with every numeric axis identical, so only the gate
can be doing the work.

|log2(a/b)|, combined with L-infinity

Scale-free, symmetric, zero at equality, inf when one side is zero. Burstiness
uses |log2((1+a)/(1+b))| because D = 0 is a real trace and a bare ratio would
make a perfectly paced trace incomparable to everything, including another paced
one.

Six axes. The p95s are in because the p50s alone cannot separate a
long-context workload from a short one with a heavy tail — which is exactly where
prefix caching and chunked prefill live.

L-inf, not a mean, asserted on the case it exists for: identical on five axes,
8× off on input_p95 reads 0.5 as a mean and 3.0 as L-inf. The mean
applies the row. For scale the two real traces are 4.929 apart, limited by
input_p95.

rate_rps is deliberately not an axis, and the exclusion is material rather
than an omission: two regimes identical but for an 8× rate difference are 0.0
apart by default and 3.0 with the axis on. The inclusion rule and the
experiment that would settle it are written down.

No threshold ships

AxisTolerance(max_distance=1.0)                                              # ValidationError
AxisTolerance(max_distance=1.0, calibration="prereg E4: sign flip at 1.4")   # ok

An exact regime match still returns a row, so the schema is usable today; any
nonzero distance returns UNCALIBRATED, names the limiting axis, and routes
to discovery. A placeholder cannot quietly become a production constant. The
four-step calibration procedure is in the module, so the open state has an exit.

Lifecycle

A row is refused at construction without provenance: no knobs, repeats < 2, a
delta missing its latency percentiles, an empty trace_sha256. MeasuredDelta
carries throughput and TTFT/ITL percentiles — the schema cannot express a
throughput-only row. Invalidation is a field with a reason, never a deletion.

evidence ∈ {measured, illustrative} exists because the worked examples ship in
the same format as real rows; without it an example copied into a live playbook is
indistinguishable from a promoted one.

row_from_runs builds a row from two joined runs — a BenchRun is one arm, a
row is the difference — refusing arms that did not run the same trace, any run
that did not reconcile, unequal repeat counts, and a missing latency percentile.
Medians, never means.

delta_is_floor

BurstGPT has no prefix identity, so a replay synthesizes unique blocks per
request: lengths hold, and no sharing is invented that the source never had. A
prefix-cache knob measured there saw the least reuse the real traffic could
have had, so its delta is a lower bound — usable as "at least this much",
never quotable as the gain. A computed property, not a stored flag.

ex6 is that case; ex2 is the control — same synthesized trace, a knob that does
not depend on reuse, so not a floor.

The examples ship nothing selectable

6 rows, 0 selectable. Regimes real, deltas invented, every row labelled. A
perfectly matching query returns nothing and names the reason per row. The
largest claimed delta in the file (+22%) is the scoreboard row — the one most
likely to be copied, gated out by equality rather than distance. Regenerated by
make_examples.py and byte-identical on re-run.

Verification

python -m gitm.traffic  --selftest    24 checks, 3 real traces, 7 drop reasons
python -m gitm.playbook --selftest    19 checks, 2 real regimes, 0 calibrated axes
pytest tests/test_traffic.py tests/test_playbook.py -q          45 passed
ruff check gitm/traffic gitm/playbook                  All checks passed!

Assertions live once and run from both entry points, so the check a reader is told
about and the check CI runs are the same check.
test_every_check_is_registered fails if a check_* function is defined and left
out of the list.

Six fixtures pinned by sha256 through the existing gitm.bench.manifest
(gitm.bench.manifest/v1), plus the real bench serve result JSON — 7 files,
150,941 bytes, verify_manifest clean. Both trace fixtures are unmodified
prefixes of published files, with their source URLs recorded.

Full suite: 11 failures, all pre-existing — module-for-module identical to the
list on clean b2da5b6, all missing optional deps or importer goldens. None in
gitm/traffic or gitm/playbook.

Two bugs found by running it rather than reading it

• runner.py executed a bare vllm from PATH, which fails under an
  absolute interpreter (a conda env used without activation — normal in CI and
  WSL) and, worse, could have validated one install and run another:
  check_vllm() reads importlib.metadata for this interpreter.
  vllm_executable() now derives the binary from sys.executable.
• --tokenizer was missing from the argv builder. bench serve builds a
  tokenizer from --model even though timed_trace sends pre-tokenized prompts,
  so any served name HuggingFace cannot resolve dies in
  AutoTokenizer.from_pretrained — long after the endpoint answered. Bites a stub
  server and any --served-model-name rename.

The one dependency change

vllm = ["vllm>=0.6"] → vllm = ["vllm>=0.23.0"], with the reasoning in the
file. bench serve --dataset-name timed_trace landed in bfb9ebc21 (2026-05-28),
one day after v0.22.0 shipped — established by checking the tags (v0.22.0 ❌,
v0.22.1 ❌, v0.23.0 ✅), not inferred from a changelog. Below the floor the run
dies on an argparse complaint about an unknown dataset name, which reads like a
typo in our command rather than a missing feature.

Not split into a separate [traffic] extra: the library is CPU-only except
for firing, so an extra named for it would claim you need a multi-gigabyte CUDA
install to use something that does not need one. The >=0.19.0 "conflict" was
prose in three install commands, never a packaging constraint.

Also included

gitm/_banner.py (56 lines, stdlib) and its six-line wiring into gitm/cli.py.
It is a dependency of both new CLIs. stderr, and only when sys.stdout.isatty()
— the gate asks about stdout even though the banner goes to stderr, because the
question is "is a human watching" and stdout is where the answer is.
check_banner_can_never_corrupt_stdout asserts the non-TTY case is silent and
stdout stays empty. The failure it guards is remote from its cause: a banner on
stdout surfaces as a JSON parse error in a CI step, nowhere near the banner.